Openstack database replication

ABSTRACT

Several different embodiments of a massively scalable object storage system are described. The object storage system is particularly useful for storage in a cloud computing installation whereby shared servers provide resources, software, and data to computers and other devices on demand. In several embodiments, the object storage system includes a ring implementation used to associate object storage commands with particular physical servers such that certain guarantees of consistency, availability, and performance can be met. In other embodiments, the object storage system includes a synchronization protocol used to order operations across a distributed system. In a third set of embodiments, the object storage system includes a metadata management system. In a fourth set of embodiments, the object storage system uses a structured information synchronization system. Features from each set of embodiments can be used to improve the performance and scalability of a cloud computing object storage system.

This is a continuation application of and claims priority to the U.S.patent application 13/089442 filed Apr. 19, 2011, entitled “MassivelyScalable Object Storage System,” and claims benefit of U.S. provisionalpatent application 61/450,166, filed Mar. 8, 2011, entitled “MassivelyScalable File Storage System.” This application is also related toco-pending non-provisional U.S. patent application Ser. Nos. 13/______,filed Apr. 19, 2011, and 13/______, filed Apr. 19, 2011. The entirety ofthese disclosures is hereby incorporated herein by reference.

BACKGROUND

The present disclosure relates generally to cloud computing, and moreparticularly to a massively scalable object storage system to providestorage for a cloud computing environment.

Cloud computing is location-independent computing, whereby sharedservers provide resources, software, and data to computers and otherdevices on demand. As a term, “cloud computing” describes a consumptionand delivery model for IT services based on the Internet, and ittypically involves over-the-Internet provisioning of dynamicallyscalable and often virtualized resources. This frequently takes the formof web-based tools or applications that users can access and use througha web browser as if it were a program installed locally on their owncomputer. Details are abstracted from consumers, who no longer have needfor expertise in, or control over, the technology infrastructure “in thecloud” that supports them. Most cloud computing infrastructures consistof services delivered through common centers and built on servers.Clouds often appear as single points of access for consumers' computingneeds.

As the use of cloud computing has grown, cloud service providers such asRackspace Hosting Inc. of San Antonio, Tex., have been confronted withthe need to greatly expand file storage capabilities rapidly whilemaking such expansions seamless to their users. Conventional filestorage systems and methods to expand such systems suffer from severallimitations that can jeopardize data stored in the object storagesystem. In addition, known techniques use up substantial resources ofthe object storage system to accomplish expansion while also ensuringdata safety. Finally, the centralization of data storage brings with itissues of scale. A typical local storage system (such as the hard drivein a computer) may store thousands or millions of individual files for asingle user. A cloud-computing-based storage system is designed toaddress the needs of thousands or millions of different userssimultaneously, with corresponding increases in the number of filesstored.

Accordingly, it would be desirable to provide an improved scalableobject storage system.

SUMMARY

According to one embodiment, the improved scalable object storage systemincludes a method for storing data, comprising providing a plurality ofphysical storage pools, each storage pool including a plurality ofstorage nodes coupled to a network, each storage node further providinga non-transient computer readable medium for data storage; classifying aplurality of availability zones, wherein the storage nodes within anavailability zone are subject to a correlated loss of access to storeddata; defining a plurality of abstract partitions, wherein each possibleinput data management request deterministically corresponds to one ofthe plurality of abstract partitions; mapping the plurality of abstractpartitions to the plurality of physical storage pools such that eachmapped physical storage pool includes a replica of the data associatedwith the associated mapped abstract partition, and each replica for aparticular abstract partition is mapped to a physical storage pool in adifferent availability zone; receiving a data management request overthe network, the data management request associated with a data object;identifying a first partition corresponding to the received datamanagement request; and manipulating the data object in the physicalstorage pools mapped to the first partition in accordance with the datamanagement request.

According to another embodiment, the improved scalable object storagesystem includes a distributed storage coupled to a network, thedistributed storage including a first storage pool and a second storagepool from a plurality of storage pools, the first storage pool in afirst availability zone and the second storage pool in a secondavailability zone, each storage pool including at least one processor, acomputer readable medium, and a communications interface; a directorcoupled to the network, the director including a processor, a computerreadable medium, and a communications interface; a ring structureassociated with the director, wherein the ring structure is adapted toassociate a storage request with a first abstract partition from aplurality of abstract partitions, and wherein the ring structure isfurther adapted to selectively associate a first abstract partition witha first fault-tolerant multi-master replication target, the firstreplication target including the first storage pool and the secondstorage pool; wherein the director is adapted to route inbound storagerequests to the replication target and outbound storage responses fromthe replication target.

According to another embodiment, the improved scalable object storagesystem includes a non-transient computer readable medium containingexecutable instructions, which when executed on a processor at a firsttime, initialize a ring by retrieving a set of ring parameters, the ringparameters including a number of abstract partitions, a number ofphysical storage pools, and a set of performance constraints; performinga consistent hashing function associating a first range of inputs with afirst abstract partition and a second range of inputs with a secondabstract partition; and allocating the available physical storage poolsby mapping each abstract partition to one or more storage pools inaccordance with the set of performance constraints; at a second time,opaquely route an input request to a correct storage pool in accordancewith the initialized ring; and at a third time, rebalance the ring byretrieving the set of ring parameters, performing a consistent hashingfunction associating the range of inputs with the first abstractpartition and the second range of inputs with the second abstractpartition; and allocating the available storage pools mapping eachabstract partition to one or more storage pools in accordance with theset of performance constraints such that each abstract partition haszero or one changes in the physical storage pools allocated thereto.

According to another embodiment, the improved scalable object storagesystem includes a system for coordinating events in a distributedsystem, comprising a plurality of subsidiary nodes coupled to a network,each subsidiary node including at least one processor, acomputer-readable medium, and a communications interface, whereininformation in a first subsidiary node needs to be synchronized with theinformation in a second subsidiary node in response to a time-varyingseries of requests; a first gateway, including a first processor, afirst local clock, and a first communications interface; a secondgateway, including a second processor, a second local clock, and asecond communications interface; a timekeeping node coupled to thenetwork, including a master clock; and a synchronization rectifiercoupled to the first and second subsidiary nodes; wherein thetimekeeping node is operationally coupled to the first and secondgateways to reduce clock skew between the master clock, the first localclock and the second local clock below a configurable threshold; whereinthe first gateway uses the first processor to timestamp a first requestreceived over the first communications interface according to the timeof the first local clock with a granularity at least equal to theconfigurable threshold; wherein the second gateway uses the secondprocessor to timestamp a second request received over the secondcommunications interface according to the time of the second local clockwith a granularity at least equal to the configurable threshold; whereinsynchronization between the first subsidiary node and the secondsubsidiary node is controlled by the later-occurring request if thefirst request and the second request are separated by a time greaterthan the configurable threshold; and wherein synchronization between thefirst subsidiary node and the second subsidiary node is controlled bythe synchronization rectifier if the first request and the secondrequest are separated by a time smaller than the configurable threshold.

According to another embodiment, the improved scalable object storagesystem includes a method for coordinating events in a distributedsystem, comprising synchronizing a master clock to coordinated universaltime within a master skew threshold; synchronizing a first local clockat a first gateway with the master clock within a system skew threshold,and synchronizing a second local clock at a second gateway with themaster clock within the system skew threshold; receiving, at the firstgateway, a first request to manipulate a non-volatile data storage, andmarking the first request with the time of reception according to thefirst local clock, with a granularity at least equal to the system skewthreshold; receiving, at the second gateway, a second request tomanipulate the non-volatile data storage, and marking the second requestwith the time of reception according to the second local clock, with agranularity at least equal to the system skew threshold; evaluate thefirst request and the second request to determine if they areunambiguously ordered by determining if the first request and the secondrequest are temporally ordered with a granularity greater than thesystem skew threshold; if the first request and the second request areunambiguously ordered, modifying the non-volatile data storage asdirected in the later request; and if the first request and the secondrequest are not unambiguously ordered, modifying the non-volatile datastorage as directed by a deterministic tiebreaker.

According to another embodiment, the improved scalable object storagesystem includes a non-transient computer readable medium containingexecutable instructions, which when executed on a processor synchronizea first local clock with a second local clock within a system skewthreshold; receive a first request to manipulate a system resource andmarks the first request with the time of reception according to thefirst local clock, with a granularity at least equal to the system skewthreshold; receive a second request to manipulate the system resourceand marks the second request with the time of reception according to thesecond local clock, with a granularity at least equal to the system skewthreshold; evaluate the first request and the second request todetermine if they are unambiguously ordered by determining if the firstrequest and the second request are temporally ordered with a granularitygreater than the system skew threshold; if the first request and thesecond request are unambiguously ordered, manipulates the systemresource as directed in the later request; and if the first request andthe second request are not unambiguously ordered, executes tiebreakerinstructions controlling the system resource; and returns a success orerror depending on the outcome of the tiebreaker instructions.

According to another embodiment, the improved scalable object storagesystem includes a method for managing data items in a distributedstorage pool, comprising providing a plurality of physical storagepools, each storage pool including a plurality of storage nodes coupledto a network, each storage node further providing a non-transientcomputer readable medium for data storage; storing a first replica of adata item in a first physical storage pools, and storing a secondreplica of the data item in a second physical storage pool; in responseto receiving a modification instruction for the data item, selectivelymodifying the first replica of the data item, creating a firstmodification sentinel file, and storing the first modification sentinelfile in the first physical storage pool; in response to encountering thefirst modification sentinel file during a data item replication process,modifying the second replica of the data item and creating a secondmodification sentinel file in the second physical storage pool.

According to another embodiment, the improved scalable object storagesystem includes a system for out-of-band communication of object storagemetadata, the system comprising a distributed storage system coupled toa network, the distributed storage including a first storage pool and asecond storage pool from a plurality of storage pools, the first andsecond storage pools each including at least one processor, a computerreadable medium, and a communications interface; wherein the firststorage pool includes a first replica of a data item, and the secondstorage pool includes a second replica of the data; an object serviceresponsive to modification instructions; and a replicator adapted tocreate a second replica of the data item in the second storage pool;wherein the object service responds to an out-of-band instruction byselectively modifying the first replica of the data item, creating afirst modification sentinel file, and storing the first modificationsentinel file in the first physical storage pool; and wherein thereplicator responds to encountering the first modification sentinel fileduring a data item replication process by modifying the second replicaof the data item and creating a second modification sentinel file in thesecond physical storage pool.

According to another embodiment, the improved scalable object storagesystem includes a non-transient computer readable medium containingexecutable instructions, which when executed on a processor at a firsttime, run a replication procedure that takes a first copy of a data itemin a first location and makes an identical second copy of the data itemin a second location; at a second time, run an out-of-band modificationprocedure to selectively modify the first copy of the data item, createa first modification sentinel file, and store the first modificationsentinel file in the first location; and at a third time, change theexecution of the replication procedure to modify a the second copy ofthe data item and create a second modification sentinel file in thesecond location.

According to another embodiment, the improved scalable object storagesystem includes a non-transient computer readable medium containingexecutable instructions, which when executed on a processor at a firsttime, run a replication procedure that takes a first copy of a data itemin a first location and makes an identical second copy of the data itemin a second location; at a second time, run an out-of-band modificationprocedure to selectively modify the first copy of the data item, createa first modification sentinel file, and store the first modificationsentinel file in the first location; and at a third time, change theexecution of the replication procedure to modify a the second copy ofthe data item and create a second modification sentinel file in thesecond location.

According to another embodiment, the improved scalable object storagesystem includes a distributed information synchronization system,comprising a first subsidiary node coupled to a network, the firstsubsidiary node including a first non-transitory computer-readablemedium wherein the first computer-readable medium includes a firststructured information repository, and wherein information in the firststructured information repository is subject to internal consistencyconstraints; a second subsidiary node coupled to a network, the secondsubsidiary node including a second non-transitory computer-readablemedium wherein the second computer-readable medium includes a secondstructured information repository, and wherein information in the secondstructured information repository is subject to internal consistencyconstraints; a repository synchronizer coupled to the first and secondstructured information repositories; the repository synchronizer furtherincluding a consistency evaluation module adapted to evaluate thedifferences between the first structured information repository and thesecond structured information repository; an internal modificationmodule adapted to modify the internal structures of a structuredinformation repository; an external replication module adapted to deletea target structured information repository and replace it with areplicated copy of a source structured information repository; and athreshold comparator; wherein the repository synchronizer is adapted toevaluate the first and second structured information repositories anddetermine a level of difference and compare the level of difference to aconfigurable threshold using the threshold comparator; if the level ofdifference is above the configurable threshold, modify the internalstructures of a selected structured information repository using theinternal modification module; and if the level of difference is belowthe configurable threshold, delete the selected structured informationrepository and replace it with a replicated copy of a consistentstructured information repository using the external replication module.

According to another embodiment, the improved scalable object storagesystem includes a method for synchronizing structured information in adistributed system, comprising storing a first structured informationrepository on a first non-transitory computer-readable medium, whereininformation in the first structured information repository is subject tointernal consistency constraints; storing a second structuredinformation repository on a second non-transitory computer-readablemedium, wherein information in the second structured informationrepository is subject to internal consistency constraints; evaluatingthe differences between the first structured information repository andthe second structured information repository to determine a preferredstate and a difference measurement quantifying a difference from thepreferred state; determining whether the difference measurement exceedsa configurable threshold; modifying a selected structured informationrepository if the difference measurement for the selected structuredinformation repository is less than the configurable threshold, whereinthe modification of the selected structured information repository issubject to the internal consistency constraints of the selectedstructured information repository, deleting the selected structuredinformation repository if the difference measurement for the selectedstructured information repository is greater than the configurablethreshold, and replacing the selected structured information repositorywith a replica of a structured information repository in the preferredstate, wherein either modifying the selected structured informationrepository or deleting and replacing the structured informationrepository changes the non-transitory computer-readable medium storingthe selected structured information repository such that the selectedstructured information repository is both compliant with its internalconsistency constraints and in the preferred state. The method may alsoinclude determining that both the first structured informationrepository and the second structured information repository are not inthe preferred state; pre-selecting the structured information repositorythat is closer to the preferred state and modifying the pre-selectedstructured information repository to bring the pre-selected structuredinformation repository to the preferred state, subject to the internalconsistency requirements of the pre-selected structured informationrepository, regardless of the configurable threshold.

According to another embodiment, the improved scalable object storagesystem includes a non-transient computer readable medium containingexecutable instructions, which when executed on a processor update afirst structured information repository on a first non-transitorycomputer-readable medium, subject to internal consistency constraints;update a second structured information repository on a secondnon-transitory computer-readable medium, subject to internal consistencyconstraints; evaluate the differences between the first structuredinformation repository and the second structured information repositoryto determine a preferred state and a difference measurement quantifyinga difference from the preferred state; determine whether the differencemeasurement exceeds a configurable threshold; modify a selectedstructured information repository if the difference measurement for theselected structured information repository is less than the configurablethreshold, subject to the internal consistency constraints of theselected structured information repository, delete the selectedstructured information repository if the difference measurement for theselected structured information repository is greater than theconfigurable threshold, and replace the selected structured informationrepository with a replica of a structured information repository in thepreferred state.

According to another embodiment, the improved scalable object storagesystem includes a non-transient computer readable medium containingexecutable instructions, which when executed on a processor update afirst structured information repository on a first non-transitorycomputer-readable medium, subject to internal consistency constraints;update a second structured information repository on a secondnon-transitory computer-readable medium, subject to internal consistencyconstraints; evaluate the differences between the first structuredinformation repository and the second structured information repositoryto determine a preferred state and a difference measurement quantifyinga difference from the preferred state; determine whether the differencemeasurement exceeds a configurable threshold; modify a selectedstructured information repository if the difference measurement for theselected structured information repository is less than the configurablethreshold, subject to the internal consistency constraints of theselected structured information repository, delete the selectedstructured information repository if the difference measurement for theselected structured information repository is greater than theconfigurable threshold, and replace the selected structured informationrepository with a replica of a structured information repository in thepreferred state.

The specifics of these embodiments as well as other embodiments aredescribed with particularity below.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 a is a schematic view illustrating an embodiment of a filestorage system.

FIG. 1 b is a schematic view illustrating an embodiment of aninformation handling system used in the file storage system of FIG. 1 a.

FIG. 2 is a schematic view illustrating an embodiment of a logicalstructure provided by the file storage system of FIG. 1 a.

FIG. 3 is a schematic view illustrating an embodiment of a user account.

FIG. 4 is a flow chart illustrating an embodiment of a method forstoring an object.

FIG. 5 is a flow chart illustrating an embodiment of a method forcreating a ring

FIG. 6 flow chart illustrating an embodiment of a method for reassigningpartitions in a ring.

DETAILED DESCRIPTION

Referring now to FIG. 1 a, an embodiment of a file storage system 100 isillustrated. The file storage system 100 includes a user device 102connected to a network 104 such as, for example, a Transport ControlProtocol/Internet Protocol (TCP/IP) network (e.g., the Internet.) Astorage management server 106 is connected to the network 104 and to aplurality of storage servers 108. While only one user device has beenillustrated as connected to the network 104 for clarity of discussion,one of skill in the art will recognize that a plurality of user devicesmay, and typically will, be connected to the network 104. While only onestorage management server coupled to a plurality of storage servers hasbeen illustrated as connected to the network 104 for clarity ofdiscussion, one of skill in the art will recognize that a plurality ofstorage management servers, each connected to a plurality of storageservers may, and typically will, be connected to the network 104. Eachof the user device 102 and the storage management server 106 includes arespective network interface for communicating with the network 104(e.g., outputting information to, and receiving information from, thenetwork 104).

Each of the user device 102, storage management server 106, and theplurality of storage servers 108 may include a respective informationprocessing system, a subsystem, or a part of a subsystem for executingprocesses and performing operations (e.g., processing or communicatinginformation). An information processing system is an electronic devicecapable of processing, executing or otherwise handling information.Examples of information processing systems include a server computer, apersonal computer (e.g., a desktop computer or a portable computer suchas, for example, a laptop computer), a handheld computer, and/or avariety of other information handling systems know in the art.

Referring now to FIG. 1 b, an information processing system 110 which isrepresentative of one of, or a portion of, the information processingsystems described above, is illustrated. The information processingsystem 110 may include any or all of the following: (a) a processor 112for executing and otherwise processing instructions, (b) a plurality ofinput devices 116, which are operably coupled to the processor 112, forinputting information, (c) an optional display device 116, which isoperably coupled to the processor 112, for displaying information, (d)an optional print device 118, which is operably coupled to the processor112, for printing visual images, scanning visual images, and/or faxingvisual images, (e) a computer-readable medium 120, which is operablycoupled to the processor 114, for storing information, as discussedfurther below, and (f) various other electronic circuitry for performingother operations of the information processing system 110 known in theart. For example, the information processing system 110 may include (a)a network interface (e.g., circuitry) for communicating between theprocessor 110 and the network 104 and/or other devices, and (b) a memorydevice (e.g., FLASH memory, a random access memory (RAM) device or aread-only memory (ROM) device for storing information (e.g.,instructions executed by processor 112 and data operated upon byprocessor 112 in response to such instructions)).

The computer-readable medium 120 and the processor 110 are structurallyand functionally interrelated with one another as described below infurther detail, and information processing system of the illustrativeembodiment is structurally and functionally interrelated with arespective computer-readable medium similar to the manner in which theprocessor 110 is structurally and functionally interrelated with thecomputer-readable medium 120. As discussed above, the computer-readablemedium 120 may include a hard disk drive, a memory device, and/or avariety of other computer-readable media known in the art, and whenincluding functional descriptive material, data structures are createdthat define structural and functional interrelationships between suchdata structures and the computer-readable medium 120 (and other aspectsof the system 100). Such interrelationships permit the data structures'functionality to be realized. For example, the processor 112 reads(e.g., accesses or copies) such functional descriptive material from thecomputer-readable medium 120 onto the memory device of the informationprocessing system 110, and the information processing system 110 (moreparticularly, the processor 112) performs its operations, as describedelsewhere herein, in response to such material stored in the memorydevice of the information processing system 110. In addition to readingsuch functional descriptive material from the computer-readable medium120, the processor 112 is capable of reading such functional descriptivematerial from (or through) the network 104. In one embodiment, thecomputer-readable medium is non-transitory.

Referring now to FIGS. 1 a and 2, the file storage system of FIGS. 1 aand 1 b creates a logical structure 200. The logical structure 200includes a user 202 connected to a proxy 204. In one embodiment, theuser 202 may be provided by the user device 102, the proxy 204 may beprovided by the storage management server 106, and the user 202/proxy204 connection may be created by the coupling of the user device 102 tothe storage management server 106 through the network 104. The proxy 204is connected to one or more rings 206 such as an object ring 206 a, acontainer ring 206 b, and an account ring 206 c, described in furtherdetail below, that are connected to an object service 208, containerservice 210, and an account service 212, respectively, described infurther detail below. In other embodiments, there are other types ofobjects managed by rings, such as a structured data ring, a graphstorage ring, or another type of ring (not pictured). In suchembodiments, each ring would be connected to an appropriate service,such as a structured data service, a graph service, or another service(not pictured).

Each of object service 208, the container service 210, and the accountservice 212 are connected to a plurality of storage pools 214. In oneembodiment, the rings 206 may include software that is stored on acomputer-readable medium location in the storage management server 106and/or the storage servers 108. In one embodiment, the object service208, the container service 210, and the account service 212 may includesoftware that is stored on a computer-readable medium located in thestorage management server 106 and/or the storage servers 108. In oneembodiment, the storage pools 208 may be provided by the storage servers108. In one embodiment, the proxy 204/rings 206/object service208/container service 210/account service 212/storage pool 214connections may be created by the connection of the storage managementserver 106 with the storage servers 108. In a further embodiment, therings are implemented at least in part using electrical circuits on asemiconductor chip to achieve better speed and latency.

In one embodiment, each storage pool 214 is provided by a separatestorage server 108 or includes a virtual server that is included in aportion of one of the storage servers 108 or across a plurality of thestorage servers 108. For example, the storage servers 108 may bephysically located in one or more data centers, and the resources of thestorage servers 108 may be virtualized according to the requirements ofa plurality of users (e.g., the user 202) such that the plurality ofstorage pools 214 are provided to the plurality of users in order tostore files and/or data objects. Thus, resources for a particularvirtual server or storage pool may span across multiple storage servers108.

Referring now to FIG. 3, the user 202, which is exemplary of a pluralityof users that use the file storage system 100, has a user account 300with the file storage system 100 to store and receive data objects, andthat user 202 may create a plurality of containers 302 in the useraccount 300 and store a plurality of data objects 304 in each of thecontainers 302 for retrieval. In the discussion below, a user account isreferred to as an “account”, a container is referred to as a“container”, and a data object us referred to as an “object” for clarityof discussion. One of skill in the art will recognize that the terms“account”, “container” and “object” are generic forms of data namingthat are used to direct the file storage system 100 to a specific dataobject. When other types of rings and services are used, an appropriatename may be substituted. For clarity, discussion of alternative ringsand services will be limited to the “account”, “container” and “object”rings and services.

The components of the file storage system 100 and some of theirfunctions will now be described in detail.

The Rings 206

As discussed above, the rings 206 are implemented in a tailoredelectrical circuit or as software instructions to be used in conjunctionwith a processor to create a hardware-software combination thatimplements the specific functionality described herein. To the extentthat software is used to implement the rings, it may include softwarethat is stored on a computer-readable medium location in the storagemanagement server 106 and/or the storage servers 108. Referring back toFIG. 2, the rings 206 include semiconductor circuits and/orcomputer-executable instructions that, when executed by a processor,provide subsystems of the file storage system 100 that provide a mappingbetween the entities stored in the file storage system 100 and thelocations of those entities in the storage pools 214. In the illustratedembodiment, the file storage system 100 includes a separate object ring206 a, container ring 206 b, and account ring 206 c, and when componentsof the file storage system 100 need to perform any operation on anobject, container, or account, those components interact with the objectring 206 a, container ring 206 b, and account ring 206 c, respectively,to determine the location of that stored entity in the storage pools214. However, one of skill in the art will recognize that different ringstructures may be provided (e.g., a single ring for the objects,containers, and accounts, more than one ring for each of the objects,containers, and account, etc.) without departing from the scope of thepresent disclosure. The rings 206 maintains the availability and safetyof data in the file storage system 100 through the use of zones,partitions, replicas, and the storage pools 214, as described below.

A zone is defined as one or more of the storage pools 214 that aresubject to a correlated loss of access or data as a result of aparticular event. For example, each storage server 108 in the filestorage system 100 may be defined as a storage pool in a separate zone,as each storage server 108 is subject to loss of access to its storedobjects as a result of a storage device failure, a catastrophic event atthe location where the storage server resides, and/or a variety of otherobject access-loss scenarios known in the art. For the same reasons, adrive in a storage server 108 may be defined as a storage pool in aseparate zone, a plurality of storage servers 108 in a given storagerack or cabinet as a storage pool in a separate zone, a plurality ofstorage servers 108 coupled to the same switch as a storage pool in aseparate zone, a plurality of storage servers 108 in a given datacenteras a storage pool in a separate zone, a plurality of storage servers 108connected to a common power system as a storage pool in a separate zone,etc. One of skill in the art will recognize that the examples of zonesprovided above are not limiting, and a variety of zones known in the artwill fall into the scope of the present disclosure.

Logically, a partition is an abstract storage bucket. As discussed infurther detail below, the file storage system 100 maps each partition toa plurality of storage pools 214 that are in different zones, and storesdata using those partitions. The mapping of a given partition to aplurality of storage pools 214 creates a plurality of partition replicasof that partition (e.g., equal to the number of storage pools 214 thepartition is mapped to.) For example, when a given partition is mappedto 3 storage pools 214 that are in different zones, 3 partition replicasof that partition are created.

The object ring 206 a for the management of objects will be described indetail below. However, one of skill in the art will recognize how thediscussion may be applied to the container ring 206 b, the account ring206 c, and/or a ring for any other stored entity, without departing fromthe scope of the present disclosure.

In various replicated, network-based file storage systems, an objectfrom a user is received by a proxy. To determine where the object shouldbe stored, some attribute of the object or the object data itself ishashed. If necessary, some attribute of the object is modified so thatthree different results are returned from the hashing function. Theobject is then replicated and stored in the storage pool correspondingto the number returned by the hash function.

Under typical circumstances, a consistent hashing function is used asthe hash function. The use of the consistent hashing function ensuresthat there will be minimal changes to the assigned storage pools given achange in membership due to adding or removing new storage pools.

Although the consistent hashing function results in minimal changes tothe storage location, sometimes the assignments made by the consistenthashing function or the rearrangements needed due to a change inmembership may have undesirable storage characteristics. For example,such methods have been found to result in multiple object replicas forthe same object being stored in one or more storage pools that are inthe same zone. As discussed above, this is undesirable because thenmultiple (and possibly all) object replicas for the same object aresubject to being lost as a result of a particular event. Alternatively,rebalancing the replicas due to a change in membership has been found torequire the movement to two of the replicas 4% of the time, and themovement of all three replicas 1% of the time. It is desirable to neverhave to move more than one replica at a time.

In one embodiment, the file storage system 100 solves the problem ofmultiple object replicas for the same object being stored in storagepools that are in the same zone through the use of the rings 206.Referring now to FIG. 4, a method 400 for storing stored entities isillustrated. At block 402, an object us received by a user. In oneembodiment, an object is received from the user 202 by the proxy 204.The method 400 then proceeds to block 404 where a partitionidentification is generated. In one embodiment, a consistent hashfunction is applied to the object received in block 402, and the hashfunction returns a partition identification that corresponds to apartition. The method 400 then proceeds to block 406 where the partitionassociated with the partition identification is mapped to storage poolsthat are in different zones. This mapping function is constrained sothat the physical location of the storage pools is required to have oneor more desirable properties, such as having each partition replica in aseparate zone.

There are various embodiments of the constrained mapping function. Inone embodiment, the constrained mapping function is the output of aconstraint satisfaction solver, in which the desired storagecharacteristics (such as the requirement that each replica of apartition be in a different availability zone) are inputs to the solvingfunction. The solver then uses one or more search methodologies withinthe solution space to find a storage layout that maps partitions tostorage pools in a desirable manner.

In a second embodiment, a constrained mapping function is applied toportions of the partition identification (e.g., the portions of thepartition identification that the constrained mapping function isapplied to) may be bits of the output of the original hashing functionis applied to the object. For example, the number of bits to which theconstrained mapping function is applied may be known as the partitionpower, and 2 to the partition power may indicate the partition count.The constrained mapping function is designed to return a storage poollocation for each portion of the partition identification to which it isapplied, and the storage pool locations returned for a given partitionidentification will each correspond to storage pools 214 in differentzones. These storage pool locations are then associated with thepartition identification. Thus, the partition corresponding to thepartition identification is replicated multiple times in the filestorage system 100 (i.e., a partition replica is included in eachstorage pool corresponding to the storage pool locations determined fromthe constrained mapping function.) The method 400 then proceeds to block408 where the object is stored according to the partition. The objectreceived by the user 202 in block 402 of the method 400 may then bestored according to the partition corresponding to the partitionidentification, which results in multiple object replicas for the objectbeing stored in storage pools that are in different zones in the filestorage system 100. In another embodiment, the constrained mappingfunction is used to determined storage pool locations that are indifferent zones for each partition prior to the object being received bythe user 202, discussed in further detail below.

The output of the constrained mapping function signifies a particularstorage pool where a replica of the partition should be stored. Anexample of this output is as follows: When an object is received fromthe user 202 at block 402 of the method 400, and at block 404 of themethod 400, a hash function is applied to the object. In one exemplaryembodiment, the user 202 provides data including anaccount/container/object name to the proxy 2004, and a hash function isapplied to the account/container/object name as follows:

Hash function (account/container/object name)==123456789Where 123456789 is the partition identification that is returned by thehash function. At block 406 of the method 400, the partition mappingnumber may then be divided into 3 parts (e.g., the first three digits,the second three digits, and the third three digits of the partitionidentification,) and the constrained mapping function is applied to eachof those parts:Constrained mapping function (123)==storage pool location (zone 1)Constrained mapping function (456)==storage pool location (zone 7)Constrained mapping function (789)==storage pool location (zone 3)

As discussed above, the constrained mapping function is designed toreturn the storage pool location (zone 1), storage pool location (zone7), and storage pool location (zone 3) that correspond to storage poolsthat are in different zones (e.g., zones 1, 3, and 7). The storage poolslocations are then associated with the partition identification:

Partition identification: (storage pool location (zone 1)), storage poollocation (zone 7), storage pool location (zone 3))

Thus, the partition corresponding to the partition identification isreplicated across storage pools that are in different zones (here, zones1, 3, and 7.) At block 408 of the method 400, the object received fromthe user 202 is then stored, using the partition corresponding to thepartition identification, in each of the storage pools corresponding tothe storage pool locations returned by the application of theconstrained mapping function to portions of the partitionidentification. Thus, 3 replicas of the object received from the user202 are stored in the file storage system 100 in storage pools that arelocated in different zones (zones 1, 3, and 7.) In one embodiment, eachof the storage pool locations are IP addresses, i.e., when each of thestorage pools are separate storage servers. In one embodiment, theconstrained mapping function is a hash function. However, one of skillin the art will recognize that a variety of functions may be used toensure that each partition is mapped to storage pools that are indifferent zones without departing from the scope of the presentdisclosure.

In another embodiment, the constrained mapping function is applied tothe file storage system 100 before the object is received by the user202 at block 402 in order to accomplish the mapping of the partitions tostorage pools described above with reference to block 406 of the method400. For example, the total number of partitions and the total number ofstorage servers/storage pools in the file storage system 100 may (andtypically will) be known. With that knowledge, the constrained mappingfunction is used to map each partition in the file storage system 100 toa plurality of storage pools that are in different zones, and thatinformation is stored in a constrained mapping database. For example, aconstrained mapping database may include partitions mapped to storagepools such as:

Partition 1: (storage pool location (zone 1)), storage pool location(zone 2), storage pool location (zone 3))Partition 2: (storage pool location (zone 4)), storage pool location(zone 5), storage pool location (zone 6))Partition 3: (storage pool location (zone 7)), storage pool location(zone 8), storage pool location (zone 9))

In one embodiment, the output of the constrained mapping function can besaved for optimized lookup. For example, the saved output may beembodied in a file provided to each of the storage pools 214, or storedin a database that is available for the appropriate systems to query. Ifthe saved output is contained within a file, the storage pools 214 maythen periodically check the modification time of this file and reloadtheir in-memory copies of the ring structure as needed.

Thus, when an object is received from a user 202 at block 402, the hashfunction is applied to that object to get the partition identification(e.g., partition 1, 2, or 3 in the example above) at block 404, and thenat block 406, the partition identification may then be used with theconstrained mapping database to determine the corresponding partitionand its associated storage pool locations. This embodiment allows theprocessing necessary to map partitions to multiple storage pools indifferent zones to be conducted before objects are received from usersso that such processing does not have to be conducted each time anobject is received from a user.

For example, referring now to FIG. 5, a method for building a ring 206is illustrated. At block 502, an ideal number of partitions for eachstorage pool in the file storage system is determined. In oneembodiment, the number of partitions that should ideally be assigned toeach storage pool 214 is calculated based the weight (e.g., storagecapacity) of each storage pool 214. For example, if the partition poweris 20, the ring 206 will have 1,048,576 (2²⁰) partitions. If there are1,000 storage pools 214 of equal weight, each storage pool 214 willideally be assigned 1,048.576 partitions. This may be referred to as anideal partition count, and in the example, each storage pool 214 startsoff empty with a 1,048.576 ideal partition count. The method 500 thenproceeds to block 504 where the storage pools are placed in a sortingorder. In one embodiment, the storage pools 214 are placed in a sortingorder based on their ideal partition count (e.g., highest to lowest),and this sorting order is maintained throughout the method 500 aspartitions are assigned storage pools 214. The method 500 then proceedsto block 506 where partitions are assigned to storage pools based ontheir sorting order but with a zone restriction. In one embodiment, thepartitions are assigned to the storage pool 214 with the highest idealpartition count, but subject to the restriction that the storage pool214 to which a partition is being assigned is not in the same zone asany other storage pool 214 that includes a partition replica for thatsame partition. The method 500 then proceeds to block 508 where thesorting order of the storage pools is adjusted. In one embodiment, oncea partition is assigned to a storage pool 214, that storage pool 214will have its ideal partition count decremented and thus that storagepool 214 is moved to a lower position in the sorting order, and themethod 500 then returns to block 506 to continue to assign partitions tostorage pools based on their sorting order but with the zonerestriction. In such a manner, each partition is assigned multiplestorage pools 214 in different zones, and thus objects received fromusers may have multiple object replicas stored in storage pools indifferent zones simply by associating those objects with the partitions.

As mentioned above, another problem relates to the rebalancing of objectreplicas stored in the file storage system due to changing membership(i.e., adding or subtracting storage servers or storage pools from thefile storage system.) Such methods have been found to require the movingof multiple object replicas of the same object in response to amembership change, which is undesirable.

In one embodiment, the mapping of partitions to multiple storage poolsin different zones in the file storage system 100 described above solvesthese problems. The use of the constrained mapping function to ensurethat each partition is mapped to storage pools in different zonesensures that object replicas for the same object are never located instorage pools 214 that are in the same zone (i.e., because any givenobject received from a user is stored in a partition that is replicatedin storage pools that are in different zones.) For example, with eachstorage server 108 defined as a separate zone, the addition orsubtraction of a given storage server 108 from the file storage system100 thus can only effect one partition replica, and hence one objectreplica of a given object (i.e., because only one of the partitionreplica will ever be located on a storage server that is defined as aseparate zone.) In similar fashion, the rebalancing associated withchanging the zone membership can be accomplished without affecting morethan one replica because each zone is guaranteed to only contain onereplica of a given partition.

Periodically, partitions may need to be reassigned to different storagepools 214, and the reassignment of partitions will result in thebuilding of a new ring from an old ring. Such an event may occur due tothe removal and/or addition of a storage pool 214 from the file storagesystem 100 (e.g., a membership change.) Referring now to FIG. 6, amethod 600 for reassigning partitions in response to the removal of astorage pool is illustrated. The method 600 begins at block 602 wherethe ideal number of partitions for each storage pool is recalculated. Inone embodiment, the ideal partition count for the storage pools 214remaining in the file storage system 100 (subsequent to the removal of astorage pool) are recalculated. The method 600 then proceeds to block604 where the storage pools are placed in a sorting order as describedabove with reference to block 504 of the method 500. The method thenproceeds to block 606 where partitions to be reassigned are grouped. Inone embodiment, a partition list for the partitions to be reassigned iscreated. For example, any storage pools 214 that have been removed fromthe file system 100 may have all their assigned partitions unassignedand added to the partition list, and any storage pools 214 that havemore partitions than their ideal partition count may have randompartitions unassigned from them and added to the partition list (i.e.,such that those storage pools have a number of partitions that is withina predetermined amount of their ideal partition count.) The partitionson the partition list may then be reassigned to the storage pool 214 inblocks 608 and 610 of the method 600 substantially as discussed abovewith reference to blocks 506 and 508 of the method 500. In oneembodiment, at block 608 of the method 600, whenever a partition isreassigned to a storage pool 214, the time of the reassignment isrecorded. Reassignment times may be used when gathering partitions toreassign to storage pools 214, such that no partition replica for agiven partition is moved twice in a predetermined amount of time.However, such reassignment restrictions based on reassignment times maybe ignored for partition replicas on storage pools 214 that have beenremoved from the file storage system 100, as removing a storage pool 214only happens upon storage pool 214/storage server 108 failure and thusrequires the reassignment of the partitions.

In one embodiment, the method 600 is conducted periodically to helpbalance the amount of data stored by storage pools 214 in the filestorage system 100. For example, the partition reassignment method 600discussed above may repeated until each storage pool 214 is within apredetermined threshold of a predetermined storage capacity (e.g.,within 1% of 60% storage capacity for that storage pool) or when it isdetermined that partition reassignment will not improve the balance ofdata stored by the file storage system 100 by more than a predeterminedamount. For example, if a first storage server 108 includes 2 TB ofstorage, a second storage server 108 includes 4 TB of storage, and athird storage server 108 includes 6 TB of storage, data balancing may beconducted to ensure that each of the storage servers 108 holds the samepercentage of its storage capacity (i.e., the first storage server 108holds 0.66 TB of data, the second storage server 108 holds 1.33 TB ofdata, and the third storage server 108 holds 2 TB of data such that eachof the storage servers 108 is at 33% of its storage capacity.) Weightsmay be applied to storage servers 108 to balance the distribution ofdata on the storage servers 108 in the file storage system 100 toaccount for different storage capacities.

Object Service 208

As discussed above, the object service 208 is implemented in a tailoredelectrical circuit or as software instructions to be used in conjunctionwith a processor to create a hardware-software combination thatimplements the specific functionality described herein. To the extentthat one embodiment includes computer-executable instructions, thoseinstructions may include software that is stored on a computer-readablemedium located in the storage management server 106 and/or the storageservers 108. The object service 208 may include instructions that, whenexecuted by a processor, provide object storage and objectionmanipulation functionality such that the object service 208 is operableto, for example, store, retrieve and delete stored objects in thestorage pools 214. In one embodiment, a object service 208 is providedfor each storage pool that holds object data. For example, an objectservice 208 may be included on a server that further includes one ormore storage drives that provide a storage pool for objects. In oneembodiment, the objects are stored as binary files with metadata storedas extended attributes of the file in the file system used by the objectstorage service. In such an embodiment, the object service 208 will usesthe extended attributes of the filesystem to manage the metadata. In asecond embodiment, the metadata is stored in a machine-readable formatnext to the data itself. For example, the metadata for a file is storedin a text file or single file database.

In one embodiment, objects are stored by the object service 208 using apath derived by applying a hash function to the name of the object alongwith a timestamp. For example, an incoming object for a user account tobe written to a container will have a hash applied to itsaccount/container/object name and the path generated for the object is:

/objects/<partition>/<storage pool location>/objectname_hash.15673.datawhere “objects” indicate that the object data is stored in an objectstorage pool 214, <partition> is the partition identification that mapsthe object to a partition, <storage pool location> is the storage poollocation that maps the partition replica to a storage pool 214 in adifferent zone than its related partition replicas, objectname_hash isthe hash of the account/container/object name, and 15672 is thetimestamp.

When there is a request for an object, the file storage system 100 willfind all the object replicas in the file storage system 100 that includethe objectname_hash and return the object data that has the most recenttimestamp value. Special care is needed to record updates that should bepersisted as the new canonical value. For example, when an objectreplica is deleted, a modification sentinel (e.g., a 0 byte “tombstone”file or “.ts” file) is written to the storage pool 214 where the deletedobject replica was located and that includes the same objectname_hash asthe deleted object replica (i.e., /objectname_hash.15784.ts,) and thattombstone file stays in the file storage system 100 for a predeterminedamount of time (e.g., 7 days.) During object replication, discussed infurther detail below, when the file storage system 100 encounters atombstone file, the file storage system 100 checks whether the tombstonefile has been in the system for 7 days. If not, the file storage system100 searches for and deletes any object replicas that it finds relatedto that tombstone file (e.g., replicas that same objectname_hash as thetombstone file) to ensure that objects that were meant to be deletedfrom the file storage system 100 are removed and older versions ofobject replicas of a given object do not appear in the file storagesystem 100 due to, for example, the temporary failure of a storageserver 108 or storage pool 214 that might have prevented the deletion ofthat object replica previously. If the file storage system 100determines that a tombstone file has been in the file storage system 100for longer than the predetermined time, that tombstone file is deleted.

The mechanism used for recording file deletion is also used to recordother types of updates. For example, a “purge” marker indicates that thesystem should overwrite all copies of the object and set the space tofree; a “version” marker indicates that the system should create a copyand mark the copy with a version number; and a “ttl” (time-to-live)marker indicates that the system should check an authoritative sourcefor updates after the expiry of a set time period. Other types ofout-of-band changes to the file are also contemplated.

Container Service 210

As discussed above, the container service 210 is implemented in atailored electrical circuit or as software instructions to be used inconjunction with a processor to create a hardware-software combinationthat implements the specific functionality described herein. To theextent that one embodiment includes computer-executable instructions,those instructions may include software that is stored on acomputer-readable medium located in the storage management server 106and/or the storage servers 108. The container service 210 may includeinstructions that, when executed by a processor, provide containerstorage and container manipulation functionality such that the containerservice 210 is operable to store, retrieve and delete stored containersin the storage pools 214. In one embodiment, an container service 210 isprovided for each storage pool that holds container data. For example, acontainer service 210 may be included on a server that further includesone or more storage drives that provide a storage pool for containers,and the container service 210 may include the names of containers andobjects in those containers. Thus, In one embodiment, the containerservice 210 handles the listing of containers, and does not hold thelocation where the objects are stored (e.g., the storage pool where agiven object replica resides), but rather the locations of containersthat hold the objects. The listings for the container locations may bestored as database files, and those listings may be replicated acrossthe storage pools 214 in a manner that is similar to the replication ofobjects (i.e., through their association with partitions,) discussedabove. Container storage statistics for the container service(s) 210 maybe tracked by the file storage system 100 and may include total numberof objects stored by one or more containers, the total storage providedby any given container, and/or a variety of other statistics known inthe art.

Account Service 212

As discussed above, the account service 212 is implemented in a tailoredelectrical circuit or as software instructions to be used in conjunctionwith a processor to create a hardware-software combination thatimplements the specific functionality described herein. To the extentthat one embodiment includes computer-executable instructions, thoseinstructions may include software that is stored on a computer-readablemedium located in the storage management server 106 and/or the storageservers 108. The account service 212 may include instructions that, whenexecuted by a processor, provide account storage and accountmanipulation functionality such that the account service 212 is operableto store, retrieve and delete stored accounts in the storage pools 214.In one embodiment, an account service 212 is provided for each storagepool that holds account data. For example, a account service 212 may beimplemented by a server that includes storage drives that provide astorage pool for accounts, and the account service 212 may include thenames of accounts and containers in those accounts. Thus, the accountservice 212 is very similar to the container service 210, discussedabove, with the exception that account storage 212 handles the listingsof accounts.

Other Services

As discussed above, other types of services may be implemented insimilar fashion to the object, container, and account services describedabove. For example, one implementation includes an authorizationservice. The authorization service may include instructions that, whenexecuted by a processor, handle the storage and manipulation ofauthorization metadata so that the authorization service is operable tostore, retrieve, delete, and query stored credentials from in thestorage pools 214. In one embodiment, an authorization service providesan ACL-based authorization. In a second embodiment, the authorizationservice provides posix-compatible authorization. In a third embodiment,the authorization service provides tree or graph-based authorization,such as would be provided with an LDAP-based authorization service.

A second implementation includes a structured data service. Thestructured data service may include instructions that, when executed bya processor, provide handle the storage and manipulation of structureddata such that the structured data service is operable to store,retrieve, delete, and query tabular, graph, or tree-based data from inthe storage pools 214. In one embodiment, an structured data serviceprovides a JSON-based output. In a second embodiment, the structureddata service provides XML-based output. In a third embodiment, thestructured data service provides HTML output.

Proxy 204

The proxy 204 is implemented in a tailored electrical circuit or assoftware instructions to be used in conjunction with a processor tocreate a hardware-software combination that implements the specificfunctionality described herein. The proxy 204 is responsible for tyingtogether the file storage system 100. For each request received from auser, the proxy 204 determines the location of the account, container,or object in the appropriate ring 206 (e.g., the object ring 206 a, thecontainer ring 206 b, or the account ring 206 c,) and routes the requestaccordingly. A public Application Programming Interface (API) may beexposed to users through the proxy 204. A large number of failures maybe handled by the proxy 204. For example, if a storage server 108 and/orstorage pool 214 is unavailable for a object PUT, the proxy 204 may usethe rings 206 to determine an appropriate storage server 108 and/orstorage pool 214 for that object and route the object there instead. Inone embodiment, when objects are streamed to or from a storage server108, they are streamed directly through the proxy 204 and proxy server106 to or from the user 202 and/or user device 102, and are not spooledby the proxy 204 and the proxy server 106.

In another embodiment, there are multiple proxies associated with a filestorage service. The existence of multiple proxies may be ascertainablefrom outside the file storage service, or it may be transparent.Different proxies may be used for different purposes. For example, inone embodiment different proxies are used for different types of files.In another embodiment, different proxies are used for different types ofrequests. In a third embodiment, an appropriate proxy is chosen tominimize latency, geographic, or network distance between the proxy andthe system making the request.

In one embodiment, one of the functions performed by the proxy istime-stamping or logging all requests into the storage system. Thetimestamps on the incoming requests are stored as metadata and are usedin part to determine the most recent version of a file.

In an embodiment with more than one proxy, it is possible for more thanone request to come in within a short period of time. In that case, itis important to resolve any conflicts associated with multiplesimultaneous accesses to the file storage system. In one embodiment, analgorithmic approach for ordering the actions of different independentactors is used, such as the use of a vector clock. In a secondembodiment, an independent arbiter is used to resolve conflicts, usingan implementation of the Paxos algorithm or the Byzantine Generalsalgorithm.

A third embodiment may also be used to simplify and increase the speedof the system by reducing the applicable error window and thenalgorithmically picking a “winner” in the case of multiple conflictingaccesses within the error window. For example, an embodiment may use atime synchronization server and time synchronization code to reduce theclock skew between different computers in a pool, in a zone, or acrosszones to within a specified ε, for example one millisecond or onemicrosecond. The applicable ε can be determined by analyzing the patternof accesses over time, and it may be different for different accounts,different types of services, or at different times. In this case, thealgorithmic complexity of absolutely ordering the actions across thecluster is traded for the operational complexity of handling timesynchronization across the system.

Given the clock synchronization across the pools or zones within ε, thepreviously-discussed timestamp will be sufficient to absolutely orderthe actions of other systems interacting with the file storage system ifthere are not multiple conflicting accesses to the same stored valuewithin ε. In that case, the timestamping is used to order the actionsand pick the most recent version of any information or data.

If there are multiple conflicting accesses within ε, then the systemincludes a synchronization rectifier that algorithmically breaks the tieand chooses a winner. In one embodiment, this is handled by assertingthat the first copy to replicate (as further discussed below) wins. In asecond embodiment, the inconsistency is handled by throwing an error formanual intervention. In a third embodiment, the inconsistency is handledby examining one or more additional types of metadata, such as latencyto the originating user (given identical arrival times, the originatingserver with lower latency issued second), internal file modification orcreation times, or an arbitrary ordering on an unrelated value. A fourthembodiment uses geolocation on the requesting IP address and allowingthe request that is geographically closer to the timestamping gateway. Afifth embodiment detects the attempted simultaneous access and does notallow any write, instead returning an error in response to bothrequests. A sixth embodiment evaluates the requests to see if they canbe merged and both accesses granted.

Replicators

Replicators are implemented in a tailored electrical circuit or assoftware instructions to be used in conjunction with a processor tocreate a hardware-software combination that implements the specificfunctionality described herein. To the extent that one embodimentincludes computer-executable instructions, those instructions may beimplemented as an software that is stored on a computer-readable mediumlocated in the storage management server 106 and/or the storage servers108, and may include instructions that, when executed by a processor,keep the file storage system 100 in a consistent state in the face oftemporary error conditions like network outages, storage pool 214failure, and/or storage server 108 failure. For example, an objectreplicator may be provided for each storage pool 214 (e.g., a storageserver 108 that provides a storage pool) that holds object data. Thereplicators compare stored entities in their storage pool 214 with eachreplica of that stored entity in other storage pools 214 in the filestorage system 100 to ensure that all related replicas contain thelatest version of the stored entity. In one embodiment, objectreplicators may use a hash list to quickly compare subsections ofpartitions, while container replicators and account replicators may usea combination of hashes and shared storage account metadata. In oneembodiment, replicator updates of stored entities are push based. Forexample, replicators may compare the replica stored entities in theirstorage pools 214 with related replica stored entities in other storagepools in the file storage system 100, and if the replicator determinesthere is a difference between the replicas (e.g., by applying an orderindependent check sum to the related replicas), the replicator may thenpush the data that related replica stored entities in other storagepools need in order to be up to date. In one embodiment, the pushedupdates include rsyncing replicas to efficiently provide only the dataneeded by the out-of-date replica. Account and container replicators mayeither push missing data over HTTP or rsync whole database files in theevent it is determined that a push update will be inefficient. Thepush-based updates discussed above results in replicas being updatedgenerally only from “local” storage pools 214 to “remote” storage pools214. In one embodiment, this provides a benefit as data in a storagepool 214 may not belong there (as in the case of handoffs and ringchanges), and a replicator can't know what data exists elsewhere in thefile storage system 100 that it should pull in. Thus, it's the duty ofany replicator associated with a given a storage pool 214 that containsdata to ensure that data gets to other storage pools where it belongs.As discussed above, replicators may also ensure that data is removedfrom the system by creating the tombstone files as the latest version ofa replica when that replica is deleted, and then search out and removingall replicas related to that tombstone file from the file storage system100.

Database Replicators

Database replicators are a type of replicator, discussed above, thatoperate on storage pools 214 that contain accounts or containers (i.e.,there may be account replicators and container replicators.) To performthe replication discussed above, the first step that a databasereplicator may perform may be a low-cost hash comparison to find outwhether or not two replicas (e.g., a replica on the database replicatorslocal storage pool 214 and a related replica on a “remote” storage pool214) already match. Under normal operation, the hash comparison allowsrelatively quick verification that databases in the file storage system100 are already synchronized. If the hashes differ, the databasereplicator may bring the databases in sync by sharing records addedsince the most recent previous sync point. This most recent previoussync point notes the last record at which two databases were known to bein sync. After all new records have been pushed to the remote database,the sync table (which lists which remote databases a local database isin sync with) of the local database is pushed to the remote database, sothe remote database knows it's now in sync with database that the localdatabase has previously synchronized with. If a database replica (e.g.,an account replica or container replica) is found to be missing entirelyfrom a storage pool 214 that it should exist in, the entire localdatabase file may be recreated on that storage pool 214 using rsynctechniques known in the art. In one embodiment, when an entire localdatabase file is be recreated on a storage pool 214 using rsync, thatdatabase may be vested with a new unique id.

Object Replicator

Object replicators are a type of replicator, discussed above, thatoperate on storage pools 214 that contain objects. In one embodiment,object replicators associated with a storage pool 214 may performedrsync techniques known in the art on remote storage pools to determineappropriate data to push data to remote storage pools. However, asobject replication times may increase using this method when the filestorage system 100 gets sufficiently large, a hash of the contents foreach suffix directory may instead be saved to a per-partition hashesfile, and the hash for a given suffix directory is then invalidated whenthe contents of that suffix directory are modified. The objectreplicator may then read these hash files, calculate any invalidatedhashes, and transmit the hashes to each remote storage pool 214 thatshould hold the partition, and only suffix directories with differinghashes on the remote server are then rsynced. After pushing data to theremote storage pools 214, each rsynced suffix directory has its hashesrecalculated. Object replicator performance is generally bound by thenumber of uncached directories it has to traverse, usually as a resultof invalidated suffix directory hashes. In one embodiment, the filestorage system 100 is designed so that around 2% of the hash space on anormal storage pool 214 will be invalidated per day.

Updaters

Updaters are implemented in a tailored electrical circuit or as softwareinstructions to be used in conjunction with a processor to create ahardware-software combination that implements the specific functionalitydescribed herein. To the extent that one embodiment includescomputer-executable instructions, those instructions may includesoftware that is stored on a computer-readable medium located in thestorage management server 106 and/or the storage servers 108, and mayinclude instructions that, when executed by a processor, process updatesthat may have failed. An updater may be provided with each storage pool(e.g., on a server that includes the storage pool) to process failedupdates. For example, there may be times when container or account datawill not be immediately updated. Such incidents may occur during failurescenarios or periods of high load. If an update of a stored entityfails, the update is queued in a storage pool 214 on the file storagesystem 100, and the updater that is associated with that storage pool214 will process the failed updates. In such situations, a consistencywindow is used. For example, suppose the container service 210 is underload and a new object is put in to the file storage system 100. Theobject will be immediately available for reads as soon as the proxy 204responds to the user 202 that the object has been successfully added tothe file storage system 100. However, due to the heavy load, a containerservice 210 may not have been able to update its object listing, and sothat update would be queued for a later update. Container listings,therefore, may not immediately contain the object, although the objecthas been saved and replicated within the applicable object storage poolarea. In one embodiment, the consistency window needs only to be aslarge as the frequency at which the updater runs.

Auditors

Auditors are implemented in a tailored electrical circuit or as softwareinstructions to be used in conjunction with a processor to create ahardware-software combination that implements the specific functionalitydescribed herein. To the extent that one embodiment includescomputer-executable instructions, those instructions may includesoftware that is stored on a computer-readable medium located in thestorage management server 106 and/or the storage servers 108, and mayinclude instructions that, when executed by a processor, check theintegrity of the objects, containers, and accounts stored in the storagepools 214. If corruption is found (in the case of bit rot, for example),auditors may quarantine the file, and then replication (discussed above)is used to replace the bad file from another replica. If other errorsare found they may be logged (for example, an object's listing can't befound on any container storage that it should be on).

Large Object Support

The file storage system 100 may include a limit on the size of a singleuploaded object (e.g., 5 GB.) However, the download size of a singleobject may be made virtually unlimited with the concept of segmentation.Segments of a larger object are uploaded and a special manifest file iscreated that, when downloaded, sends all the segments, concatenated inorder, to emulate a single object. This also offers much greater uploadspeed by enabling the parallel upload of different segments. Forexample, a user may specify the segment size to use when splitting alarge file (e.g., 1 GB.) The file storage system 100 would then splitthe large file into 1 G segments and begin uploading those segments inparallel. Once all the segments have been uploaded, a manifest file maycreated so the segments can be downloaded as one. In one embodiment, allthe segments may be uploaded into a second container with names likelarge_file/1290206778.25/21474836480/00000000,large_file/1290206778.25/21474836480/00000001, etc. In one embodiment,the use of a separate container provides a benefit that the maincontainer listings will not be polluted with all the segment names. Theuse of a segment name format of <name>/<timestamp>/<size>/<segment>provides that an upload of a new file with the same name won't overwritethe contents of the first until the last moment when the manifest fileis updated.

The primary factor driving the limitation of object size in the filestorage system 100 is maintaining balance among the partitions of thering. To maintain an even dispersion of disk usage throughout the filestorage system 100, larger objects are split into smaller segments whichare then glued together during a read. This design allows the user tosupport backup and archiving of large data sets, and improvesperformance and/or reduces errors due to network interruption. The majordisadvantage of this method is that knowledge of the originalpartitioning scheme is required to properly reassemble the object, whichis not practical for some use cases, such as CDN origination. In orderto eliminate any barrier to entry for clients wanting to store objectslarger than 5 GB, a fully transparent support for large object uploadsmay be provided. A fully transparent implementation supports a largermax size by automatically splitting objects into segments during uploadwithin the proxy without any changes to the client API. All segments arecompletely hidden from the client API. The “user manifest” designdiscussed above provides a transparent download of large objects to theclient and still provide the uploading client a clean API to supportsegmented uploads. Alternative “explicit” user manifest options may beprovided that include a pre-defined format for listing the segments to“finalize” the segmented upload.

Account Reapers

An account reaper is implemented in a tailored electrical circuit or assoftware instructions to be used in conjunction with a processor tocreate a hardware-software combination that implements the specificfunctionality described herein. To the extent that one embodimentincludes computer-executable instructions, those instructions mayinclude software that is stored on a computer-readable medium located inthe storage management server 106 and/or the storage servers 108, andmay include instructions that, when executed by a processor, remove datafrom deleted accounts in the background of the file storage system 100.An account may be marked for deletion by a user which may put a value ofDELETED into the status column in the account service 212 on a storagepool 214 that includes accounts, indicating the data for the accountshould be deleted later. In one embodiment, there may be no setretention time and no undelete, as it may be assumed the user willimplement such features once it is truly desired that the account's databe removed. The account reaper runs on each account service and storagepool 214 and scans the storage pool 214 occasionally for accounts thatare marked for deletion. In one embodiment, the account reaper may onlytrigger on accounts that storage pool 214 is designated as a “primary”storage pool for so that multiple account services aren't trying to dothe same work at the same time. The deletion process works as follows:for each container in the account, each object is deleted and then thecontainer is deleted. Any deletion requests that fail won't stop theoverall process, but may cause the overall process to fail eventually(for example, if an object delete times out, the container won't be ableto be deleted later and therefore the account won't be deleted either).The overall process continues even on a failure so that it doesn't gethung up reclaiming file storage system space because of one troublesomespot. The account reaper will keep trying to delete an account until iteventually becomes empty, at which point the database replicator willeventually remove the database files.

In one embodiment, deleting an account is accomplished throughcompletely external calls and requires no changes to the file storagesystem 100. All data is simply deleted in the same way a user would,through the public ReST API. However, this uses proxy resources and logseverything when such actions aren't necessary, and may require adedicated system for issuing the delete requests. In one embodiment, acompletely bottom-up approach may be used where the object and containerservers occasionally scan the data they hold, check if the account hasbeen deleted, and removing the data if the account has been deleted.This provides for the quick reclamation of storage space with no impacton the proxies or logging, but a vast majority of the scanning resultsin no action while creating a relatively high I/O load with no benefit.In one embodiment, the account server marks all the containers fordeletion and the container storage deletes the objects in each containerand then themselves. This also provides for quick reclamation of storagespace for accounts with a lot of containers, but results in a relativelylarge load spike. In one embodiment, the load spike may be reduced byslowing down the process, but then the benefit of quick reclamation ofstorage space is lost while providing a more complex process. In oneembodiment, a database replicator scans all the containers for thosemarked for deletion while performing a replication scan.

Ring Data Structure

The list of available areas in the storage pool 214 is known internallyto the Ring. In one embodiment, each item in the list of storage pool214 is a dictionary with the following keys:

id integer The index into the list storage pools 214. zone integer Thezone the storage pool 214 resides in. weight float The relative weightof the storage pool 214 in comparison to other storage pools 214. Thisusually corresponds directly to the amount of disk space the storagepool 214 has compared to other storage pools 214. For instance a storagepool 214 with 1 terabyte of space might have a weight of 100.0 andanother storage pool 214 with 2 terabytes of space might have a weightof 200.0. This weight can also be used to bring back into balance astorage pool 214 that has ended up with more or less data than desiredover time. A good average weight of 100.0 allows flexibility in loweringthe weight later if necessary. ip string The IP address(es) of thestorage server(s) 108 providing the storage pool 214. port int The TCPport the listening storage server process uses that serves requests forthe storage pool 214. device string The on disk name of the storage pool214 on the storage server(s). For example: sdb1 meta string Ageneral-use field for storing additional information for the storagepool 214. This information isn't used directly by the storage serverprocesses, but can be useful in debugging. For example, the date andtime of installation and hardware manufacturer could be stored here.

In one embodiment, the list of storage pools 214 contains holes, orindexes set to None, for storage pools 214 that have been removed fromthe cluster. Generally, storage pool ids are not reused. Also, somestorage pools 214 may be temporarily disabled by setting their weight to0.0.

Partition Assignment List

This is a list of array(‘I’) of storage pool ids. The outermost listcontains an array(‘I’) for each replica. Each array(‘I’) has a lengthequal to the partition count for the ring. Each integer in thearray(‘I’) is an index into the above list of storage pool 214.

Partition Shift Value

The partition shift value is known internally to the Ring. This valueused to shift an MD5 hash to calculate the partition on which the datafor that hash should reside. Only the top four bytes of the hash is usedin this process.

In one embodiment, a “live ring” option may be used where each storageserver 108 maintains its own copy of the ring and the storage servers108 use a gossip protocol to communicate when changes made. In oneembodiment, all ring lookups are done by calling a service on a separateserver or set of servers. In one embodiment, servers submit changerequests to a ring server to have a new ring built and shipped back outto the servers. The ring 206 described above has each replica of apartition independently assigned to a storage pool 214. In oneembodiment, a ring may be created that uses a third of the memory of thering 206 by assigning the first replica of a partition directly whiledetermining the other two replicas by “walking” the ring untiladditional storage pools 214 are found in other zones. In anotherembodiment, partition to storage pool 214 assignments are not stored ina big list in memory but rather each storage pool 214 is assigned a setof hashes or anchors. The partition is then determined from the dataitem's hash and the nearest storage pool 214 anchors determine where thereplicas should be stored.

Various hashing algorithms may be used in different embodiments. TheSecure Hash Algorithm (SHA) offers relatively good security but may beslower. MurmurHash may be used as a relatively faster option as comparedto SHA. MD5 Hash is desirable for its general availability, gooddistribution, and adequate speed.

Although illustrative embodiments have been shown and described, a widerange of modification, change and substitution is contemplated in theforegoing disclosure and in some instances, some features of theembodiments may be employed without a corresponding use of otherfeatures. Accordingly, it is appropriate that the appended claims beconstrued broadly and in a manner consistent with the scope of theembodiments disclosed herein.

1. A distributed information synchronization system, comprising: a firstsubsidiary node coupled to a network, the first subsidiary nodeincluding a first non-transitory computer-readable medium wherein thefirst computer-readable medium includes a first structured informationrepository, and wherein information in the first structured informationrepository is subject to internal consistency constraints; a secondsubsidiary node coupled to a network, the second subsidiary nodeincluding a second non-transitory computer-readable medium wherein thesecond computer-readable medium includes a second structured informationrepository, and wherein information in the second structured informationrepository is subject to internal consistency constraints; a repositorysynchronizer coupled to the first and second structured informationrepositories; the repository synchronizer further including: aconsistency evaluation module adapted to evaluate the differencesbetween the first structured information repository and the secondstructured information repository; an internal modification moduleadapted to modify the internal structures of a structured informationrepository; an external replication module adapted to delete a targetstructured information repository and replace it with a replicated copyof a source structured information repository; and a thresholdcomparator; wherein the repository synchronizer is adapted to: evaluatethe first and second structured information repositories and determine alevel of difference and compare the level of difference to aconfigurable threshold using the threshold comparator; if the level ofdifference is above the configurable threshold, modify the internalstructures of a selected structured information repository using theinternal modification module; if the level of difference is below theconfigurable threshold, delete the selected structured informationrepository and replace it with a replicated copy of a consistentstructured information repository using the external replication module.2. The system of claim 1, wherein the differences between the first andsecond structured information repositories are caused by changes overtime.
 3. The system of claim 1, wherein both the first and secondstructured information repositories include differences, and whereinrepository synchronizer uses the internal modification module tosynchronize the first information repository and uses the externalreplication module to synchronize the second information repository. 4.The system of claim 1, wherein the structured information repositoriesare structured as on-disk databases.
 5. The system of claim 4, whereinthe databases are relational.
 6. The system of claim 4, wherein thedatabases are structured as one of a tree and a graph.
 7. The system ofclaim 1, wherein the external replication module uses the rsyncprotocol.
 8. The system of claim 1, wherein the internal consistencyconstraints consist of an order-independent checksum.
 9. A method forsynchronizing structured information in a distributed system,comprising: storing a first structured information repository on a firstnon-transitory computer-readable medium, wherein information in thefirst structured information repository is subject to internalconsistency constraints; storing a second structured informationrepository on a second non-transitory computer-readable medium, whereininformation in the second structured information repository is subjectto internal consistency constraints; evaluating the differences betweenthe first structured information repository and the second structuredinformation repository to determine a preferred state and a differencemeasurement quantifying a difference from the preferred state;determining whether the difference measurement exceeds a configurablethreshold; modifying a selected structured information repository if thedifference measurement for the selected structured informationrepository is less than the configurable threshold, wherein themodification of the selected structured information repository issubject to the internal consistency constraints of the selectedstructured information repository, deleting the selected structuredinformation repository if the difference measurement for the selectedstructured information repository is greater than the configurablethreshold, and replacing the selected structured information repositorywith a replica of a structured information repository in the preferredstate, wherein either modifying the selected structured informationrepository or deleting and replacing the structured informationrepository changes the non-transitory computer-readable medium storingthe selected structured information repository such that the selectedstructured information repository is both compliant with its internalconsistency constraints and in the preferred state.
 10. The method ofclaim 9, wherein the difference measurement measures changes over time.11. The method of claim 10, wherein the preferred state is determined byexamining all the changes to the first structured information repositoryup to the time of evaluation.
 12. The method of claim 10, wherein thepreferred state is determined by including all the changes to multiplestructured information repositories up to the time of evaluation. 13.The method of claim 9, wherein the selected structured informationrepository is the first structured information repository, and thestructured information repository in the preferred state is the secondstructured information repository.
 14. The method of claim 9, the methodfurther comprising: after evaluating the differences between the firststructured information repository and the second structured informationrepository, but before determining whether the difference measurementexceeds a configurable threshold: determining that both the firststructured information repository and the second structured informationrepository are not in the preferred state; pre-selecting the structuredinformation repository that is closer to the preferred state andmodifying the pre-selected structured information repository to bringthe pre-selected structured information repository to the preferredstate, subject to the internal consistency requirements of thepre-selected structured information repository, regardless of theconfigurable threshold.
 15. The method of claim 9, wherein replacing theselected structured information repository with a replica of astructured information repository is done via the rsync protocol. 16.The method of claim 9, wherein the internal consistency constraintsconsist of an order-independent checksum.
 17. A non-transient computerreadable medium containing executable instructions, which when executedon a processor: update a first structured information repository on afirst non-transitory computer-readable medium, subject to internalconsistency constraints; update a second structured informationrepository on a second non-transitory computer-readable medium, subjectto internal consistency constraints; evaluate the differences betweenthe first structured information repository and the second structuredinformation repository to determine a preferred state and a differencemeasurement quantifying a difference from the preferred state; determinewhether the difference measurement exceeds a configurable threshold;modify a selected structured information repository if the differencemeasurement for the selected structured information repository is lessthan the configurable threshold, subject to the internal consistencyconstraints of the selected structured information repository, deletethe selected structured information repository if the differencemeasurement for the selected structured information repository isgreater than the configurable threshold, and replace the selectedstructured information repository with a replica of a structuredinformation repository in the preferred state.
 18. The non-transientcomputer readable medium of claim 17, further including instructionsthat determine that both the first structured information repository andthe second structured information repository are not in the preferredstate; pre-select the structured information repository that is closerto the preferred state and modify the pre-selected structuredinformation repository to bring the pre-selected structured informationrepository to the preferred state, subject to the internal consistencyrequirements of the pre-selected structured information repository,regardless of the configurable threshold.
 19. The non-transient computerreadable medium of claim 17, wherein the instructions modifying theselected structured information repository, when executed, perform aseries of set operations on the selected structured informationrepository.
 20. The non-transient computer readable medium of claim 17,wherein the instructions modifying the selected structured informationrepository, when executed, perform a series of graph operations on theselected structured information repository.